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SUMMARY  PAGE 


THE  PROBLEM 

A  computer  program  designed  to  aid  the  Independent  Duty  (8402) 
Corpsman  in  diagnosis  and  management  of  abdominal  pain  has  been  placed 
aboard  submarines  for  test  and  evaluation*  A  parallel  evaluation  of  the 
program's  efficacy  when  used  by  physicians  in  the  emergency  room  of  a 
Naval  hospital  is  reported  here. 


THE  FINDINGS 

Percentage  agreement  between  initial  diagnosis  and  final  diagnosis 
was  about  the  same  whether  the  source  of  the  initial  diagnosis  was  physician 
judgment  or  computer  calculation  based  on  physician-supplied  data.  Accuracy 
of  both  sets  of  initial  diagnoses  was  greatest  for  cases  ultimately 
diagnosed  as  "non-specific  abdominal  pain."  The  relatively  small  number 
of  patients  otherwise  afflicted  precludes  any  firm  conclusion  regarding 
initial  diagnoses  in  those  cases. 


APPLICATIONS 

The  program  seems  "medically  safe"  when  it  concludes  with  high  probability 
that  the  patient  has  presented  with  non-specific  abdominal  pain. 


ADMINISTRATIVE  INFORMATION 

This  report  was  submitted  for  review  in  February  1985 ,  and  was  approved 
for  publication  in  March.  It  is  designated  as  NAVSUBMEDRSCHLAB  Report 
Number  1003. 
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Abstract 


In  July  1982  the  Naval  Submarine  Medical  Research  Laboratory 
launched  a  five-year  study  aboard  nuclear  submarines  of  the 
computer-assisted  diagnostic  program  for  abdominal  pain.  Concurrently, 
a  similar  study  with  physician-collected  data  at  the  Emergency  Room 
at  the  Naval  Hospital,  Groton,  CT,  was  begun.  A  total  of  90  cases  of 
abdominal  pain  (male  and  female)  was  collected.  Approximately  80%  of 
these  had  a  final  diagnosis  of  non-specific  abdominal  pain.  The 
program  seems  ’’medically  safe”  when  this  category  is  predicted  with  a 
high  probability.  Due  to  the  limited  number  of  cases  collected, 
however,  the  program  is  not  validated  for  this  category  or  the  other 
five  categories  of  abdominal  pain. 

Further  data  collection  is  anticipated  and  desirable.  Over  a 
five-year  period  sufficient  data  should  be  collected  to  permit  at 
least  a  partial  validation  of  both  systems. 
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EVALUATION  OF  A  COMPUTER-ASSISTED  DIAGNOSIS  PROGRAM 
FOR  ACUTE  ABDOMINAL  PAIN  WITH  PHYSICIAN-COLLECTED  DATA 


I.  BACKGROUND 

The  Naval  Submarine  Medical 
Research  Laboratory  (NSMRL)  is 
tasked  with  developing  new  methods 
and  procedures  for  assisting  the 
medical  department  representative 
(MDR)  aboard  deployed  United  States 
Navy  submarines.  Paramount  among 
the  duties  of  the  MDR  is  the  eval¬ 
uation  and  treatment  of  crewmembers 
who  are  ill.  The  history  and 
physical  exam  are  the  cornerstone 
of  the  evaluation ,  with  laboratory 
facilities  being  modest.  Consulta¬ 
tion  with  other  medical  personnel 
is  attempted  only  in  emergencies 
and  if  the  mission  permits  it. 

With  the  knowledge  that  computer- 
assisted  diagnosis  was  under  study 
elsewhere,  NSMRL  saw  an  opportunity 
to  develop  a  computerized  diagnostic 
aid  for  use  by  the  MDR.  Since 
abdominal  pain  was  the  most  frequent 
chief  complaint  in  medical 
evacuations  from  submarines, 
development  of  a  computer  program 
to  assist  in  differential  diagnosis 
of  this  condition  was  initiated 
several  years  ago.  The  intent  was 
(and  is)  to  provide  a  "sophisticated 
decision  aid"  to  supplement  the 
modest  laboratory  facilities  and 
to  compensate  for  the  lack  of  a 
consultant . 

A  prototype  program  was  formu¬ 
lated  in  conjunction  with  Dr.  F. 

T.  de  Dombal  of  Leeds  Hospital  in 
the  United  Kingdom.  It  was 
modified  for  the  submarine 
population  and  made  ready  for  sea 
trials.  Proper  scientific  and 
medical  protocol  required  that 
the  project  be  conducted  as 
an  experiment  and  that 


a  control  (c)  and  experimental  (E) 
group  be  formed.  A  protocol  was 
approved  by  the  Committee  for  the 
Protection  of  Human  Subjects  which 
involved  informed  consent  for 
participation  from  MDR's  and  their 
randomization  into  a  control  and 
experimental  group.  The  Control 
group  has  the  datasheet  ,  reference 
manual,  and  (recently)  the  training 
program.  A  five-year  study  period 
was  projected  from  the  anticipated 
case  load  so  that  statistical 
requirements  could  be  met.  The 
case  load  was  estimated  from 
incidence  data  for  abdominal  pain 
supplied  by  MDR?s.  General 
descriptions  of  the  program  have 
been  previously  reported.1-5 

In  January  1982,  approval  was 
granted  for  Fleet  T&E  support.  In 
May  19825  training  of  Squadron 
Medical  Officers  and  Squadron 
Corpsmen  was  accomplished  with 
concomitant  distribution  of 
computer  tapes  and  instructional 
materials.  Individual  submarine 
corpsmen  were  then  trained  by 
Squadron  Medical  personnel.  In 
July  1982,  sea  trials  were 
initiated.  Concurrent  with  the 
sea  trials ,  NSMRL  conducted  an 
intra-laboratory  investigation  of 
the  diagnostic  program  using 
physician-collected  data.  The 
purpose  was  to  provide  an 
additional  assessment  of  the 
efficacy  of  the  system.  It  was 
not  expected  that  this  study 
would  rigorously  validate  the 
database.  This  report  summarizes 
the  findings  of  that  investigation. 
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II .  METHODS 

Using  the  abdominal  pain  data¬ 
sheet  previously  described2 , 
physician-students  and  physician- 
graduates  of  the  Naval  Undersea 
Medical  Institute  collected  data 
when  assigned  to  duty  at  the  Naval 
Hospital  Groton  (NHGRTN)  Emergency 
Room  from  May  1982-April  1984.  The 
protocol  was  explained  to  all 
physicians  and  the  datasheet 
definitions  reviewed.  Physicians 
were  asked  to  complete  a  datasheet 
on  all  patients  with  a  chief 
complaint  of  abdominal  pain, 
severity  unspecified.  Patients  old 
enough  to  relate  a  clear  history 
(about  age  5)9  and  older,  were 
included  as  were  female  patients 
who,  in  the  opinion  of  the 
practitioner,  had  a  non-gynecologic 
source  of  pain.  Datasheets  had 
to  be  completed  preferably  at  the 
time  of  the  visit  but  certainly 
within  48  hours.  On  occasion,  the 
emergency  treatment  record  (ETR) 
was  used  by  the  physician  to  recall 
details  of  the  case.  If  datasheets 
were  not  completed  within  *+8  hours, 
or  if  more  than  3  of  the  38 
categories  on  the  datasheet  were 
incomplete,  then  the  data  was  not 
used. 

Approximately  ten  physician- 
graduates  stationed  full  time  in 
the  NHGRTN  area  collected  data.  On 
the  average,  they  spent  24  hours  per 
month  in  the  Emergency  Room. 
Approximately  25  physician-students 
collected  data,  spending  a  total 
of  600  physician-hours  in  the 
Emergency  Room.  Follow-up  visits 
and  calls  were  periodically  made  to 
those  collecting  data  to  remind 
them  of  the  study,  to  provide 
datasheets,  etc.  No  attempt  was 
made  to  double  check  the  number  of 
patients  with  abdominal  pain 


presenting  during  a  physician* s  duty 
hours  with  the  number  of  datasheets 
submitted.  Physicians  were  promised 
follow-up  of  the  patient* s  outcome 
and  a  computer  printout  from  the 
program  as  an  inducement  to  collect 
dat  a  * 

Datasheets  returned  to  NSMRL  were 
reviewed  for  completion  and  briefly 
compared  with  the  ETR.  Any 
discrepancies  were  reviewed  with  the 
physician  or  the  patients.  The  ETR 
was  considered  the  correct  document 
if  a  discrepancy  was  unresolved, 
although  there  were  minimal 
discrepancies  and  no  major  ones  (i.e. 
on  occasion  the  vital  signs  were 
different).  Follow-up  was 
accomplished  by  means  of  a  phone  call 
to  the  patient  at  least  7  days  after 
the  ER  visit.  The  patient  was  then 
followed  until  the  pain  resolved  or  a 
specific  diagnosis  had  been  made. 
Final  diagnosis  was  assigned  by  the 
author  based  on  the  overall  record 
and  the  phone  call.  If  the  patient 
had  been  admitted  to  the  hospital, 
then  the  discharge  diagnosis  was  used 
If  an  operation  had  been  performed, 
the  pathologist *s  diagnosis  was  used. 

After  assignment  of  final 
diagnosis,  the  data  were  entered  into 
the  computer  and  a  '’case  summary 
page"^>  was  generated.  The 
program  is  written  such  that  the 
probabilities  of  the  six  categories 
of  diagnosis  (Table  i)  total  100%. 

The  conditional  probabilities  of  the 
database  were  identical  for  males 
and  females.  The  prior  probabilities 
were  different  for  males  and  females 
(Table  II )  although  those  for  males 
were  identical  to  prior  probabilities 
used  in  the  at-sea  trials.  Both  sets 
of  prior  probabilities  were  obtained 
through  de  Dombal.* 
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Table  I 


Categories  of  Diagnosis 


Diagnostic  Category 

Abbreviation 

Appendicitis 

APPY 

Nonspecific  Abdominal  Pain 

NONSAP* 

Renal  Colic 

RCOLIC 

Perforated  Duodenal  Ulcer 

PERFDU 

Cholecystitis 

CHOLE 

Small  Bowel  Obstruction 

SMBOBS 

^Defined  as  abdominal  pain  that  is 
not  requiring  medical  evacuation. 

non-surgical,  not  life-threatening ,  and 

Table  II 

Prior  Probabilities* 

Diagnostic  Category 

Males 

Females 

Appendicitis 

.18 

.12' 

Nonspecific  Abdominal  Pain 

.70 

-75 

Renal  Colic 

.03 

.01 

Perforated  Duodenal  Ulcer 

. 001** 

.001** 

Cholecystitis 

-05 

.11 

Small  Bovel  Obstruction 

.03 

.02 

*Rounded  to  nearest  hundredth,  except,  for  PERFDU. 

**This  is  an  artificial  number  used  to  permit  calculations  while  reflecting 
the  low  prior  probability. 
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III.  RESULTS 

A  total  of  91  datasheets  with 
supporting  documentation  were 
submitted  by  10  different 
practitioners.  Three  of  these 
practitioners  accounted  for  66 
cases.  Two  cases  were  submitted  by 
physicians  assistants  (unsolicited) 
and  were  reviewed  by  the  supervisory 
medical  officer  in  the  ER.  These 
cases  were  included  as  the  data 
were  collected  properly.  One  case 
was  discarded  due  to  inadequate 
follow-up  after  a  transfer.  A 
total  of  90  cases  were  accepted  for 
review,  51  male  cases  and  39 
female  cases.  Table  III  shows  the 
case  number  with  the  practitioner  * s 
initial  diagnosis,  the  computer 
.diagnosis,  and  the  final  diagnosis 
tabulated  for  male  cases.  The 
computer  diagnosis  has  a  calculated 
probability  of  >_  95$  unless 
otherwise  listed.  Table  IV  shows 
similar  data  for  female  cases. 

From  Table  III  (male  data) , 
concordance  between  the  practi¬ 
tioner’s  initial  diagnosis  and  the 
final  diagnosis  was  88$  ( ^5/51 ) • 
Concordance  between  the  computer 
diagnosis  and  the  final  diagnosis 
was  84$  (43/51)-  Concordance 
between  the  practitioner’s  initial 
diagnosis  and  the  computer  diagnosis 
was  88$  (45/51).  For  the  purpose 
of  this  comparison,  the  computer 
diagnosis  was  arbitrarily  taken  to 
be  that  category  with  a  calculated 
probability  greater  than  50$. 

In  the  instances  where  a  final 
diagnosis  of  OTHER  was  obtained 
(cases  9 5  23,  25,  37),  both  the 
practitioner  and  the  computer  were 
judged  to  be  incorrect.  While  the 
practitioner  had  the  option  of 
selecting  OTHER,  the  computer  could 
not  select  this  category.  One  might 


argue  that  for  cases  9  and  23  (final 
diagnoses  infectious  mononucleosis 
and  inguinal  hernia) ,  NONSAP  was 
appropriate  for  practitioner  and 
computer.  In  case  25  (final 
diagnosis  -  pneumonia),  the 
presentation  was  unusual.  The 
management  of  pneumonia  is  clearly 
different  from  that  of  NONSAP.  In 
case  37  (final  diagnosis  -  mesenteric 
adenitis),  the  distinction  between 
APPY  and  NONSAP  is  difficult.  In  a 
sense,  the  computer’s  calculation  of 
APPY  -  58$,  NONSAP  -  4l$  is  keener 
than  the  practitioner’s  selection  of 
APPY. 

Of  note  is  the  high  percentage 
of  cases  with  a  final  diagnosis  of 
NONSAP  (4l/53»  11%  of  total  cases). 

This  is  to  be  expected  as  NONSAP  has 
been  found  to  have  a  prior  probability 
of  80$  in  ER  settings.  If  these 
cases  are  taken  alone,  concordance 
between  the  practitioner’s  diagnosis 
and  the  final  diagnosis  is  97$ 

(40/4l) .  Similarly,  concordance 
between  the  computer  diagnosis  and 
the  final  diagnosis  is  90$  (37/4l). 
Concordance  between  the  practi¬ 
tioner’s  diagnosis  and  the  computer 
diagnosis  was  88$  (36/4l). 

From  Table  IV  ( female  data) 
concordance  between  the  practitioner’s 
initial  diagnosis  and  the  final 
diagnosis  was  85$  (33/39)*  Concor¬ 
dance  between  the  computer  diagnosis 
and  the  final  diagnosis  was  85$ 

(33/39)-  Concordance  between  the 
practitioner’s  initial  diagnosis^  and 
the  computer  diagnosis  was  69$  (27/39). 
The  computer  program’s  selection  with 
a  probability  greater  than  50$  was 
again  taken  as  the  computer  diagnosis. 
Case  l6  left  the  computer  T,in  error” 
as  it  calculated  NONSAP  50$,  APPY  49$. 
Cases  2  and  l8  involved'  pregnant 
women,  although  it  was  felt  by  the 
practitioners  that  the  cause  of  pain 


Table  III 


USNHGROTON 

CASE  DATA:  ABDOMINAL  PAIN  - 

-  MALE 

CASE  # 

INITIAL  D 

X 

COMPUTER  D 

X 

FINAL  D 

X 

1 

NONSAP 

AP  PEND- 57%; NONS AP - 4 1 % 

NONSAP 

2 

NONSAP 

NONSAP 

NONSAP 

3 

NONSAP 

NONSAP 

NONSAP 

4 

SMBOBS 

RCOLIC-86% ; NONSAP-13% 

RENAL  COLIC 

5 

NON SAP 

NONSAP 

NON SAP 

6 

appy 

NONSAP-86% ; APPY-13% 

NON SAP 

7 

NONSAP 

NONSAP 

NONSAP  ' 

8 

NONSAP 

NONSAP 

NONSAP 

9 

NON SAP 

NONSAP 

OTHER  (Infectious 
Mononucleosis ) 

10 

NON SAP 

APPY-94% ;NONSAP-5% 

NONSAP 

11 

NONSAP 

SMBOBS 

NONSAP 

12 

NONSAP 

NONSAP 

NONSAP 

13 

NONSAP 

NON SAP 

NONSAP 

14 

NONSAP 

NON  SAP 

NON SAP 

15 

NON SAP 

NONSAP 

NONSAP 

16 

NONSAP 

NONSAP 

NONSAP 

17 

NONSAP 

NONSAP 

NONSAP 

18 

NONSAP 

NON SAP 

NON SAP 

19 

RENAL  COLIC 

RENAL  COLIC 

RENAL  COLIC 

20 

NONSAP 

NONSAP- 87%; APPY-l 2 % 

NONSAP 

21 

NONSAP 

NONSAP- 83%; APPY- 16% 

NONSAP 

22 

NONSAP 

NONSAP 

NONSAP 

23 

NONSAP 

NONS AP- 75  % ; APPY- 2  4% 

OTHER  (Inguinal 
Hernia) 

24 

APPY 

APPY 

APPY 

25 

NONSAP 

NONSAP 

OTHER  (Pneumonia) 

26 

NONSAP 

NONSAP 

NONSAP 

27 

NONSAP 

NONSAP 

NONSAP 

28 

NONSAP 

NONSAP 

NONSAP 

29 

NONSAP 

NONSAP-85% ; APPY-12% 

NONSAP 

30 

NON SAP 

NONSAP 

NON SAP 

31 

NONSAP 

NONS AP- 7  5  % ; APPY- 2  4  % 

NONSAP 

32 

NON SAP 

NON SAP 

NONSAP 

33 

NON  SAP 

NONSAP 

NONSAP 

34 

NONSAP 

NONSAP 

NON SAP 

35 

NONSAP 

NONSAP 

NONSAP 

36 

NONSAP 

NON SAP 

NONSAP 

37 

APPY 

APPY-85% ; NONSAP-41% 

OTHER  (Mesenteric 
Adenitis) 

38 

NONSAP 

NONSAP 

NONSAP 

39 

RENAL  COLIC 

RCOLIC- 5  2  % ; NON SAP- 3 1 % 

RENAL  COLIC 

40 

NON SAP 

NONSAP 

tiONSAP 
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Table  III  Cont’d 


41 

APPY 

APPY-89  % ; NONSAP-10% 

APPY 

42 

NONSAP 

NONSAP 

NONSAP 

43 

NONSAP 

NONSAP 

NONSAP 

44 

NONSAP 

NONSAP 

NON SAP 

45 

NONSAP 

NONSAP 

NONSAP 

46 

NONSAP 

NONSAP 

NONSAP 

47 

NONSAP 

NON SAP -90% ; RCOLIC-9% 

NONSAP 

48 

APPY 

APPY-91% ; NONSAP-8% 

APPY 

49 

NONSAP 

NONSAP 

NONSAP 

50 

NON SAP 

APPY- 66%; NON SAP- 3  3  % 

NONSAP 

51 

NONSAP 

NONSAP 

NON SAP 

;  Computer 

D  is  95% 

or  greater  unless  otherwise  noted. 
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Table  IV 


USNHGROTON  CASE  DATA:  ABDOMINAL  PAIN  -  FEMALE 


CASE  # 

INITIAL  D 

COMPUTER  D  * 

X 

FINAL  D 

X 

1 

NONSAP 

NONSAP 

NONSAP 

2 

CHOLECYSTITIS/ 

NONSAP 

“NONSAP 11  -PREGNANCY 

PREGNANCY 

3 

NONSAP 

NONSAP-67% ; CHOLE-30% 

NONSAP 

4 

NONSAP 

NONSAP 

NONSAP 

5 

CHOLECYSTITIS 

NONSAP-51%; CHOLE-48% 

CHOLECYSTITIS 

6 

PYELONEPHRITIS 

CHOLECYSTITIS 

PYELONEPHRITIS 

7 

NONSAP 

NONSAP 

NONSAP 

8 

NONSAP 

NONSAP 

NONSAP 

9 

APP Y /ADENITIS 

NONSAP-80% } APPY-19% 

NONSAP 

10 

UTI 

NONSAP 

NONSAP 

11 

NONSAP 

NONSAP 

NONSAP 

12 

NONSAP 

NONSAP 

NONSAP 

13 

NONSAP 

NONSAP 

NONSAP 

14 

NONSAP 

NONSAP 

NONSAP 

15 

CHOLECYSTITIS 

CHOLECYSTITIS 

CHOLECYSTITIS 

16 

NONSAP 

NONSAP-50%; APPY-49% 

NONSAP 

17 

RCOLIC  OR 

NONSAP 

UTI 

PART . SMBOBS 

18 

CHOLE 

NONSAP 

"NONSAP" -PREGNANCY 

19 

NONSAP 

APPY 

APPY 

20 

NONSAP 

CHOLE-90% ; NONSAP-9 % 

NONSAP 

21 

BILIARY  COLIC 

CHOLECYSTITIS 

CHOLECYSTITIS 

VS.  CHOLECYSTITIS 

22 

NONSAP 

NONSAP-74% ; APPY-25% 

NONSAP 

23 

NONSAP 

CHOLE-75%;NONSAP-24% 

NONSAP 

24 

BILIARY  COLIC 

NONSAP 

NONSAP 

25 

NONSAP 

NONSAP 

NONSAP 

26 

UTI 

NONSAP 

NONSAP 

27 

NONSAP 

NONSAP 

NONSAP 

28 

LG.  BOWEL  OBS. 

NONSAP 

NONSAP 

29 

BILIARY  COLIC 

NONSAP-75 %;CHOLE-24% 

BILIARY  COLIC 

30 

DIVERTICULITIS 

NONSAP 

NONSAP 

31 

NON SAP 

NONSAP 

NON SAP 

32 

NON SAP 

NONSAP 

NON-CYSTIC  OVARIAN 

MASS 

33 

NONSAP 

NONSAP 

NONSAP 

34 

NONSAP 

NONSAP 

NONSAP 

35 

NONSAP 

NONSAP 

NONSAP 

36 

NONSAP 

NONSAP-86  % ; CHOLE- 1 3  % 

NON SAP 

37 

NONSAP 

NONSAP 

NONSAP 

38 

NONSAP 

NONSAP 

NONSAP 

39 

NONSAP 

NONSAP 

NONSAP 

*Computer  D 

A 


is  95%  or  greater  unless  otherwise  noted. 
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was  unrelated  to  pregnancy. 

Possibly,  these  two  cases  should  be 
excluded  in  the  final  analysis. 

As  with  the  male  case  data, 
the  final  diagnosis  of  NONSAP 
accounted  for  a  majority  of  the 
cases  -  79$  (31/39).  Comparisons 
are  similar  to  those  above  if  this 
category  is  reviewed  separately. 

IV.  DISCUSSION 

There  are  an  insufficient 
number  of  cases  to  validate  the 
accuracy  of  the  diagnostics  as  a 
whole  or  any  of  the  individual 
diagnostic  categories.  An  estimate 
has  been  made  that  70  cases  per 
diagnostic  category,  or  a  total  of 
k20  cases  overall,  would  be  required 
for  validation.6 

Male  (51  cases)  and  female  (39 
cases)  databases  should  be 
validated  separately.  While  both 
databases  share  the  same  conditional 
probabilities,  the  prior 
probabilities  differ  (see  Table  II ). 
These  facts  make  a  separate 
validation  for  each  seem  advisable. 
Given  that  the  conditional 
probabilities  were  derived  from  all 
male  cases,  it  may  seem  inappropriate 
to  be  evaluating  female  cases.  This 
point  is  not  disputed,  but  it  is  an 
additional  reason  to  perform 
separate  male/female  validations. 

If  an  acceptable  diagnostic 
accuracy  were  found  using  "male" 
conditional  probabilities  and 
female  prior  probabilities,  this 
author  would  accept  the  results. 

The  key  is  to  accept  for  study 
female  patients  who,  in  the  initial 
estimate  of  the  practitioner,  have 
a  non-gynecologic  source  of  pain. 

The  accuracy  of  the  program 
seems  good  when  the  final  diagnosis 


is  NONSAP  and  when  the  computer 
predicts  NONSAP.  The  program  is  both 
specific  and  sensitive  in  this  regard. 
The  number  of  cases  when  either 
the  final  diagnosis  or  computer 
diagnosis  is  NONSAP  is  sufficient 
(40+)  to  conclude  that  the  program 
is  medically  "safe"  for  this 
category.  It  is  not  stastically 
validated  though,  and  thus  has  not 
passed  rigorous  scrutiny  yet. 

The  program  may  "perform" 
slightly  better  with  male  case  data 
than  with  female  case  data.  This  is 
possibly  seen  in  the  comparison  of 
the  practitioner 1 s  initial  diagnosis 
and  the  computer  diagnosis.  With 
male  cases  this  was  88#  (^5/5l) 
while  with  female  cases  it  was  69$ 
(27/39).  Although  not  reflected 
in  comparisons  with  the  final 
diagnosis,  the  program  has  a 
disquieting  effect  when  there  is 
more  frequent  disagreement  with  the 
practitioner 1 s  diagnosis . 
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